Imaging apparatuses for use in mobile apparatuses need to have both a high resolution and a small size. A reduction in size of the imaging apparatus is limited by the size and focal length of an imaging optical lens and the size of an imaging element.
In general, since the index of refraction is different depending on the wavelength of light, a scene including information of all wavelengths cannot be imaged on an image capture surface using a single lens. Therefore, typical imaging apparatuses have an optical system composed of a plurality of lenses superposed together so that light having red, green and blue wavelengths are imaged on the same imaging surface. In this structure, the optical system of the imaging apparatus is unavoidably long, so that the imaging apparatus is thick. Therefore, a multi-eye type imaging apparatus including single lenses having a short focal length has been proposed as a technique effective for a reduction in size of imaging apparatuses, particularly for a reduction in thickness (e.g., Patent Document 1).
A multi-eye type color imaging apparatus has an imaging optical system composed of a lens for light having a blue wavelength, a lens for light having a green wavelength, and a lens for light having a red wavelength. These lenses are arranged on a plane. An imaging region is provided for each lens.
In the imaging region, not only a plurality of imaging elements may be arranged, but also by a single imaging element may be divided into a plurality of regions. In this structure, the wavelength of light handled by each lens is limited, so that a single lens can be used to image an object onto an imaging surface, thereby making it possible to reduce the thickness of the imaging apparatus significantly.
FIG. 19 illustrates a schematic perspective view of a major portion of an exemplary conventional multi-eye type imaging apparatus. 1900 indicates a lens array in which three lenses 1901a, 1901b and 1901c are formed integrally. 1901a is a lens which handles light having a red wavelength. A subject image formed by the lens 1901a is converted into image information by an imaging region 1902a which has a red wavelength separation filter (color filter) attached to a light receiving portion. Similarly, 1901b indicates a lens which handles a light having a green wavelength, and an imaging region 1902b converts the light into green image information. 1901c indicates a lens corresponding to light having a blue wavelength, and an imaging region 1902c converts the light into blue image information.
These images can be superposed and combined to obtain a color image. Note that the number of lenses does not have to be limited to three, and a plurality of images having the same color may be obtained and combined.
Thus, the multi-eye type imaging apparatus can have a thin thickness. However, when images having respective colors are simply superposed and combined, the resolution of the image is determined by the number of pixels of each separated color image. Therefore, the resolution is poor, compared to typical imaging apparatuses having a Bayer array in which green, red and blue filters are arranged in a staggered pattern.
There is a technique called “pixel shift” for improving the resolution of an imaging apparatus. FIG. 20 is a conceptual diagram for explaining how to improve a resolution using the pixel shift technique. FIG. 20 is an enlarged view of a portion of an imaging element. As illustrated in FIG. 20A, the imaging element includes an optical-to-electrical conversion portion 2101 (hereinafter referred to as an “optical-to-electrical conversion portion”) which converts received light into an electrical signal, and an invalid portion 2102 (hereinafter referred to as an “invalid portion”), such as a transfer electrode or the like, which cannot convert light into an electrical signal. In the imaging element, the optical-to-electrical conversion portion 2101 and the invalid portion 2102 constitute one pixel. The pixels typically are formed regularly at predetermined intervals (pitches). A portion enclosed with a thick line in FIG. 20A is one pixel, and P indicates one pitch.
An outline of pixel shift performed using such an imaging element will be hereinafter described. Initially, an image is captured at a position of the imaging element illustrated in FIG. 20A. Next, as illustrated in FIG. 20B, the imaging element is shifted in a slanting direction (by ½ of a pixel both in the horizontal direction and in the vertical direction) so that the optical-to-electrical conversion portion 2101 of each pixel is shifted to the invalid portion 2102, before an image is captured. Thereafter, these two captured images are combined, taking into consideration the shift amount of the imaging element, as illustrated in FIG. 20C.
Thereby, a signal can be captured from the invalid portion from which a signal cannot be captured originally. Specifically, the imaged state of FIG. 20C has the same resolution as that when an imaging element having a double number of optical-to-electrical conversion portions is used to pick up an image, as compared to when the imaging element of FIG. 20A is used to perform imaging once. Therefore, if image shift is performed as described above, an image equivalent to one captured using an imaging element having a double number of pixels can be obtained without increasing the number of pixels in the imaging element.
Note that a method for improving the resolution is not limited to the above-described slanting direction shift. When shifting is performed in the horizontal direction or in the vertical direction, the resolution can be improved in the shift direction. For example, when shifts are combined in the vertical direction and in the horizontal direction, the resolution can be increased by a factor of four. In addition, the pixel shift amount is not necessarily limited to 0.5 pixels. By performing a fine pixel shift so that the invalid portion is interpolated, the resolution can be improved further.
Also, although, in the above-described example, a relative positional relationship between the imaging element and incident light is changed by shifting the imaging element, the pixel shift method is not limited to this. For example, the optical lens may be shifted instead of the imaging element. Alternatively, for example, another method has been proposed in which a parallel plate is employed (e.g., Patent Document 1). In the invention of Patent Document 1, an image which is formed on an imaging element is shifted by tilting the parallel plate.
Although the resolution can be improved by such a pixel shift, a plurality of images are captured in time series, and thereafter, are combined to generate a high-resolution image in this pixel shift. Therefore, if images that should interpolate each other are deviated from each other, the resolution may be deteriorated. Therefore, in order to combine a plurality of images captured in time series into a high-resolution image, it is necessary to eliminate a shake caused by the imaging apparatus being moved during capturing an image due to a camera-shake or the like (hereinafter referred to as a “apparatus shake”), and a shake of a subject caused by movement of the subject (hereinafter referred to a “subject shake”).
Therefore, it is essential to eliminate or correct a shake occurring in the pixel shift in order to employ the pixel shift technique to compensate for a reduction in resolution, which is a drawback of the multi-eye type that is adopted so as to achieve a small size and a thin thickness.
Some methods of eliminating a shake to the extent possible and some conventional techniques of correcting a shake have been proposed. One method is to capture an image while fixing a camera using a tripod or the like. This method can reduce an influence of an apparatus shake.
Another method is to detect and correct an apparatus shake using a shake detecting means, such as an angular velocity sensor or the like. A correction method of using both this apparatus shake correcting mechanism and the pixel shift mechanism has been proposed (e.g., Patent Document 2 and Patent Document 3).
In the invention of Patent Document 2, a shake detecting means is used to detect a shake amount, and based on the shake amount, a pixel shift direction and a pixel shift amount are corrected, and thereafter, an imaging element is shifted (pixel shift). Thereby, an influence of an apparatus shake can be reduced.
The apparatus shake correcting method does not have to be limited to the above-described method of shifting an imaging element. In Patent Document 3, a portion of optical lenses is moved, depending on a detected shake amount, to perform apparatus shake correction and pixel shift, thereby obtaining a similar effect. As methods of detecting a shake, various methods have been proposed, including a method of using an angular velocity sensor, such as a vibrating gyroscope or the like, a method of obtaining a motion vector by comparing images captured in time series, and the like.
As another method of reducing a shake, Patent Document 3 proposes a method of comparing a plurality of images captured in time series, selecting only images whose positional relationships are appropriately shifted due to an apparatus shake or the like and which have a relationship which can be therefore expected to improve the resolution, and combining the selected images. This method is all performed electrically, so that a mechanical mechanism for correcting an apparatus shake is not required, thereby making it possible to reduce the size of the imaging apparatus.
However, in the case of the fixing method of using a tripod or the like, for example, it is necessary for the user to always carry the tripod, so that the convenience for the user is significantly deteriorated, i.e., the method is not practical.
In the case of the methods of Patent Documents 2 and 3 in which an apparatus shake is detected using a sensor to perform apparatus shake correction and pixel shift, the sensor is newly required, a complicated optical system is required, and the like, which are disadvantageous to the reduction of size and thickness.
On the other hand, in the case of the method of Patent Document 3 in which a plurality of images captured in time series are compared to select images appropriate for combination, and the selected images are combined, a sensor does not have to be newly added. However, it is expected that an image is positioned appropriately by chance due to an apparatus shake or the like, so that the resolution is not reliably improved.
Patent Document 1: JP H6-261236 A
Patent Document 2: JP H11-225284 A
Patent Document 3: JP H10-191135 A